Speech Parameter Sequence Modeling with Latent Trajectory Hidden Markov Model

The weakness of hidden Markov models (HMMs) is that they have difficulty in modeling and capturing the local dynamics of feature sequences due to the piecewise stationarity assumption and the conditional independence assumption on feature sequences. Traditionally, in speech recognition systems, this limitation has been circumvented by appending dynamic (delta and delta-delta) components to the feature vectors. HMM-based speech synthesis systems [1] also use the joint vector of static and dynamic features as an observed vector in the training process. In the synthesis process, on the other hand, a sequence of static features is generated according to the output probabilities of the trained HMM given an input sentence by taking account of the explicit constraint between the static and dynamic features [2]. Although the HMM-based speech synthesis framework has many attractive features, one drawback is that the criteria used for training and synthesis are inconsistent. While the joint likelihood of static and dynamic features is maximized during the training process, the likelihood of only the static features is maximized during the synthesis process. This implies that the model parameters are not trained in such a way that the generated parameter sequences become optimal. To address this problem, Zen [3] introduced a variant of HMM called the “trajectory HMM,” which was obtained by incorporating the explicit relationship between static and dynamic features into the traditional HMM. This has made it possible to provide a unified framework for the training and synthesis of speech parameter sequences, however, it causes difficulty as regards parameter inference. Since the conditional independence assumption on the feature vectors is lost, efficient algorithms for training and decoding regular HMMs such as the Viterbi algorithm and the Forward-Backward algorithm are no longer applicable to the trajectory HMM. Thus, some approximations and brute-force methods are usually necessary to obtain training and decoding algorithms [3, 4]. In this paper, we propose formulating a new model called the “latent trajectory HMM.” In contrast with the conventional trajectory HMM, the present model splits the generative process of an observed feature sequence into two processes, one for a sequence of the joint vectors of static and dynamic features given HMM states and the other for an observed feature sequence given the sequence of the joint vectors. By treating the joint vector of static and dynamic features as a latent variable to be marginalized out, we obtain a probability density function of an observed feature sequence with a dif-